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ABSTRACT 


This  paper,  undertaken  as  part  of  CNA’s  Quo  Vadis 
II  project,  examines  alternative  statistical  models  for 
the  cumulative  distribution  of  cost  and  time  of  Navy 
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INTRODUCTION 


y  In  response  to  tasking  from  OP-98.  C’NA  is  undertaking  a  review  of  technology 
initiatives  that  will  support  the  development  of  Navy  systems*  into  the  middle 
of  the  21st  century.  The  ultimate  goal  of  this  effort,  Quo  Vadis  (Phase  II). 
is  to  present  OP-98  with  an  investment  strategy  that  minimizes  unevenness  in 
Navy  research  and  development  expenditures  over  time  and  is  consistent  with 
projections  of  future  R&D  funding.  \ 


Developing  such  a  strategy  requires  specification  of  a  methodology  for  pro¬ 
jecting  future  RAcD  project  costs.  Such  a  methodology  should  make  best  use  of 
historical  RkD  project  data  to  identify  historical  programs  that  are  sufficiently 
similar  to  future  projects  to  serve  as  a  basis  for  projecting  their  cost  profiles. 
Fortunately,  most  of  the  R&D  programs  being  forecast  by  members  of  the  Quo 
Yadis  panel  involve  new  generations  of  equipment  functionally  similar  to  sys¬ 
tems  already  in  being;  one  may  reasonably  suppose  that  the  R&D  cost  profiles 
for  these  new  systems  will  be  similar  to  those  of  antecedent  systems.  For  these 


future  systems,  the  basic  issue  is  which  of  the  antecedent  systems  to  use  as  a 
model,  or  perhaps  how  to  construct  an  appropriate  “average"  cost  profile  from 
historical  data. 


Specifying  models  for  cost  profiles  of  substantially  new  systems  is  more  chal¬ 
lenging.  In  these  cases,  it  will  be  necessary  to  identify  commonalities  in  the 
technological  requirements,  system  function,  and  platform  support  between  the 
new  system  and  historical  programs.  Ideally,  one  would  like  to  develop  a  the¬ 
ory  of  RicD  cost  profiles,  based  on  historical  data,  that  could  be  generalized  to 
substantially  new  situations. y 

l— This  paper  is  an  effort  to  specify  models  for  R&D  cost  profiles  of  Navy  sys¬ 
tems.  It  examines  the  following  as  alternative  methodologies  for  modelling  cost 
profiles; 

\  p, 

I j»  Average  cumulative  distributions  of  program  cost  and  time'  f 

.  f  '  f 

j»  Descriptive  statistical  models  based  on  ‘  curve/fitting  to  historical  data  , 
3  9  Theoretical  models  of  cost  profiles  estimated  using  historical  data.  , 


1.  Throughout  this  paper,  the  term  “Navv"  will  be  understood  to  mean  the  Department  of  the 
Navy.  i.e,.  the  l\  S.  Navv  and  Marine  Corps,  including  both  active-duty  and  reserve  forces. 

2.  Simple  averages  of  program  costs  bv  time  period  are  rejected  out  of  hand  for  two  reasons.  (1) 
inflation  effects  are  notoriously  difficult  to  eliminate,  and  (2)  such  averages  are  too  sensitive  to 
be  used  to  project  scale  and  duration. 


ALTERNATIVE  APPROACHES 


CUMULATIVE  COST/TIME  DISTRIBUTIONS 

Cumulative  cost /time  distributions  for  historical  R&D  projects  are  the  sim¬ 
plest  models  for  new  system  cost  profiles.  Cumulative  program  costs  are  graphed 
as  a  funtion  of  cumulative  project  time  elapsed,  both  in  percentage  terms.1  Fig¬ 
ures  1  through  3  show  sample  profiles  for  three  different  kinds  of  Navy  systems: 
torpedoes,  fighter  aircraft,  and  air-to-air  missiles  (AAMs). 

One  can  see  from  these  figures  that  R<kD  cost  profiles  differ  significantly 
among  system  types  and  often  within  syst  .m  types.  Comparing  figures  2  and 
3.  for  example,  it  is  clear  that.  Navy  fighter  aircraft  are  developed  on  a  much 
different  schedule  than  AAMs.  More  than  75  percent  of  the  total  project  cost 
for  fighters  is  expended  before  50  percent  of  the  project  time  has  elapsed,  while 
for  the  average  AAM.  less  than  50  percent  of  the  cost  is  expended  before  half  of 
the  project  time  has  elapsed.  (Torpedoes  tend  to  lie  between  these  extremes.) 


Figure  1.  Cumulative  cost/time  curve  for  torpedo  R&D 


1.  These  profiles  often  are  called  “Lorentz  curves"  in  the  economics  literature.  They  most  com- 
monlv  are  used  in  studies  of  income  distribution. 
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Figure  3.  Cumulative  cost/time  curve  for  AAM  R&D 
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It  is  tempting  to  use  the  mean  cumulative  cost/time  distribution  as  the  model 
for  development  costs  of  future  systems.1  However,  figures  1-3  suggest  that  this 
must  be  done  with  caution.  It  appears  from  figure  1  that  torpedo  development  is 
relatively  generic,  in  the  sense  that  several  recent  torpedo  development  projects 
have  had  almost  identical  cumulative  cost.,  time  distributions.  Thus,  the  mean 
profile  probably  is  a  good  choice  for  modeling  follow-on  torpedo  developments. 
On  the  other  hand,  for  fighter  aircraft  and  AAMs,  there  is  so  much  variation  in 
cumulative  profiles  around  the  mean  profile  that  it  could  be  misleading  to  use 
the  mean  to  model  future  developments.  Because  these  development  programs 
tend  to  be  more  idiosyncratic,  it  may  be  better  to  try  to  “match’’  the  new  system 
to  be  developed  to  a  single  antecedent  system. 


DESCRIPTIVE  STATISTICAL  MODELS 


Another  alternative  for  modeling  R&D  cost  profiles  is  to  estimate  descriptive 
regression  models  of  historical  cost  profiles.  The  concept  of  descriptive  regression 
can  be  represented  by  the  following  equation: 


cost  =  /(t.ime;parameters)  —  e  . 

Alternative  regression  models  are  generated  by  specifying  alternative  func¬ 
tional  forms  for  /(.).  Common  choices  are  linear  or  polynomial  models;  however, 
since  it  generally  is  preferable  to  work  with  cumulative  cost  profiles,  functional 
forms  that  represent  probability  distribution  functions  also  should  be  considered. 
Two  general  distribution  functions  (in  the  sense  that  no  symmetry  is  imposed) 
are  the  gamma  distribution, 

fu>  =  . 

Jo  i(a) 

and  the  beta  distribution. 


_1 _ 

B{p.q) 


j/p_1(i  -  y)q~ldy  . 


For  this  analysis,  the  gamma  and  beta  distributions,  and  a  fourth-degree 
polynomial,  were  used  as  forms  of  the  regression  function.  Since  the  gamma  and 

1.  This  is  essentially  how  the  NOSO  "20/20"  model  constructs  its  “family  curves." 
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beta  distribution  are  nonlinear  in  the  parameters,  simple  linear  regression  was 
not  appropriate  for  these  models.  Instead,  nonlinear  least-squares  methods  were 
used. 

Tables  1  through  3  present  estimates  for  regression  parameters  of  cumulative 
cost/ time  distributions  for  torpedoes,  fighter  aircraft,  and  AAMs.  In  each  table, 
estimates  are  presented  for  polynomial,  gamma,  and  beta  specifications  for  the 
regression  function.  Each  table  also  includes  the  standard  R 2  goodness-of-fit 
measure  for  the  alternative  specifications. 


Table  1.  Estimated  regression  parameters  for 
cumulative  cost/time  distribution  (torpedoes) 
Model/parameter  Estimated  value  Standard  error 


Polynomial  (a0  4-  o.\x  —  o2x2  -f  a3x3  —  a4x4): 


a0 

0.010 

0.027 

o-i 

-0.753 

0.403 

10.253 

1.717 

0.3 

15.182 

2.634 

a4 

i72=0.978 

6.693 

1.317 

Gamma  distribution  (/0A 

r kya-1*- 

Xydy): 

A 

9.115 

0.793 

a 

3.715 

0.311 

i?2=0.996 

Beta  distribution  (/*  Bili(j,yp  Ml 

y)q~'dy): 

P 

3.884 

0.068 

q 

6.116 

0.068 

i?2=0.989 


5 


Table  2.  Estimated  regression  parameters  for 
cumulative  cost /time  distribution  (fighter  aircraft) 
Model /parameter  Estimated  value  Standard  error 


Polynomial  (a0  —  a.\X  —  o2 x~  -f  cx^x*  -t-  a4x4): 


<*o 

-0.131 

0.150 

2.489 

1.868 

a2 

1.065 

7.011 

-5.670 

10.019 

3.270 

4.810 

i?2=0.815 

Gamma  distribution  (/o'  1e" 

Xydy ): 

A 

8.724 

2.171 

a 

2.558 

0.602 

i22=0.966 

Beta  distribution  (/0A 

y)q-ldy): 

P 

2.792 

0.131 

q 

7.208 

0.131 

J?2=0.964 


Table  3.  Estimated  regression  parameters  for 
cumulative  cost/time  distribution  (AAMs) 

Model /parameter  Estimated  value  Standard  error 

Polynomial  (a0  t  OjX  ~ 

a2x2  4-  azx 

2  -f-  0:4®^ ): 

a0 

0.070 

0.101 

-0.685 

1.190 

02 

5.927 

4.270 

03 

-6.281 

5.873 

O4 

1.953 

2.719 

i22=0.934 

Gamma  distribution  (f$ 

:  rkyQ~l£~ 

Xydy): 

A 

7.526 

1.039 

a. 

3.753 

0.496 

R2= 0.980 

Beta  distribution  (/„’  B(1pq)yp  Hi 

y)q~ldy): 

P 

4.763 

0.110 

q 

5.237 

0.110 

K 2  =0.967 


One  can  see  that  each  of  the  regression  models  fits  the  cumulative  cost /time 
distributions  quite  well.  However,  in  each  case  the  gamma  and  beta  function 
specifications  fit  somewhat  better  than  a  simple  polynomial,  with  the  gamma 
function  fitting  slightly  better  than  the  beta.  Figures  4  through  6  present  the 
raw  data  and  gamma  distribution  of  fitted  regression  functions  for  the  three 
systems. 
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Figure  4.  Descriptive  regression  data  for  torpedo  R&D 
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Figure  5.  Descriptive  regression  data  for  fighter  aircraft  R&D 


Figure  6.  Descriptive  regression  data  for  AAM  R&D 


THEORETICAL  MODELS  FOR  R&D  COST  PROFILES 

Ideally,  one  would  like  to  project  future  R&D  cost  profiles  on  the  basis  of  a 
theoretical  model  of  how  R&D  expenditures  should  be  allocated  for  the  develop¬ 
ment  of  a  new  system.  As  noted  earlier,  such  a  model  is  especially  important  if 
the  new  system  is  substantially  unlike  historical  systems,  so  there  is  no  obvious 
empirical  model  for  the  costs  of  the  new  system. 

As  a  starting  point  for  developing  such  a  model,  one  needs  to  define  the 
objective  for  spending  R&D  funds  to  develop  a  new  system.  Abstracting  from 
the  institutional  facts  of  who  actually  makes  programming  decisions,  suppose 
that  the  program  manager  needs  to  produce  N  units  of  a  system  by  some  time 
T  hence.  Presumably,  he  would  try  to  do  so  at  minimum  total  cost.1  Thus,  it 
seems  natural  to  think  of  the  problem  in  terms  of  classical  optimization:  minimiz¬ 
ing  the  objective  function  (cost)  within  the  constraints  imposed  by  production 
requirements  (JV  units)  and  available  time  (T  years). 

1.  Issues  of  discounted  costs  will  be  considered  subsequently. 
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However,  the  optimization  problem  as  posed  has  a  kind  of  intrinsic  time 
dependence.  The  answer  will  be  in  terms  of  an  optimal  time  path  of  R&D 
expenditures  (and  also  of  production  costs).  Therefore,  it  should  be  posed  as  a 
dynamic,  as  opposed  to  static,  optimization  problem,  and  it  can  be  solved  (if  at 
all)  using  optimal  control  methods.1 

Consider  the  following  (simplified)  optimal  control  problem: 
max 

mmL : c(v(t),A-,f»-*f)i* 

subject  to  =  7(f),  ^  =  g(R(t)),  Ar(0)  =  0,  and  X(T)  =  N.  The  variable 
R(t)  represents  the  time  path  of  R&D  expenditures,  K{t)  the  technical  contri¬ 
bution  to  product  purchased  by  R{t),  X(t)  the  time  path  of  system  inventory, 
and  Y(t)  the  system  production  rate.  The  Hamiltonian  for  this  control  problem 
is: 


H  =  \C(Y(i),K(t)  +  R(t))  +  A ,}'(<)  4-  A 2g(R(t))  . 

Suppose  further  that  the  functions  C{Y,K)  and  g{R(t))  have  the  following 
simple  forms: 


C(7(<).  A*(0)  =  a ln(Y(t))  -  7ln(K(i)) 
g(R(i))  =  BR(tf,0  <  3  <  1  . 

The  first,  assumption  gives  the  cost  function  a  familiar  Cobb-Douglas  form. 
The  second  assumption  implies  that  the  percentage  rate  of  growth  of  K(i )  is  less 
than  linear  in  RicD  expenditures,  so  that  increasing  RfcD  expenditures  gener’te 
increasingly  smaller  increments  to  the  rate  of  technology  growth. 

It  can  be  shown  that  the  first-order  conditions  for  a  maximum  can  be  solved 
for  the  following  optimal  time  paths: 

R'(i)  =  .4/rb 
Ym(1)  =  N/T 
X*{t)  =  (N/T)t  . 

1.  Good  basic  references  on  dynamic  optimization  are  fl;.  r2l ,  and  [3], 
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Unfortunately,  this  solution  for  R(i)  is  monotonically  declining  in  t ,  which  does 
not  correspond  very  well  to  historical  experience  across  systems.  However,  if 
the  objective  function  is  modified  to  represent  the  discounted  system  cost,  the 
solution  can  be  shown  to  follow  the  following  optimal  time  path  for  R{t): 

R’(t)  =  At^exp{-^-~t)  , 


which  has  the  correct  generic  shape.  (The  parameter  r  is  the  implicit  discount 
rate.) 

In  fact,  this  function  has  the  same  general  form  as  the  gamma  probability 
density  function  discussed  above.  This  in  turn  suggests  that  the  parameters  of 
this  simple  theoretical  model  can  be  estimated  using  the  gamma  distribution 
function  parameters  estimated  above,  which  turns  out  to  be  the  case.1  Tables  4 
through  6  present  estimates  of  the  parameters  of  this  model  for  torpedoes,  fighter 
aircraft,  and  AAMs. 


Table  4.  Estimated  regression  parameters  for 
structural  cost/time  model  (torpedoes) 


Model/parameter 

Estimated  value 

Standard  error 

,3 

0.632 

0.114 

r 

R2= 0.996 

0.173 

0.072 

Table  5.  Estimated  regression  parameters  for 


structural  cost/time  model  (fighter  aircraft) 


Model  /  parameter 

Estimated  value 

Standard  error 

3 

0.358 

0.386 

T 

f?2=0.966 

0.367 

0.346 

1.  To  avoid  problems  of  program  scale,  it  is  convenient  to  transform  the  solution  into  a  cumulative 
cost/ time  profile,  which  in  this  case  will  simpiv  be  the  ratio  of  a  gamma  probability  density 
function  and  its  associated  distribution  function. 
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Table  6.  Estimated  regression  parameters  for 
structural  cost /time  model  (A AMs) 


Model /parameter 

Estimated  value 

Standard  error 

3 

0.637 

0.180 

r 

0.228 

0.091 

R2= 0.934 

Estimates  of  the  the  discount  rate  and  the  rate  at  which  R&D  investments 
contribute  to  the  accretion  of  project-specific  technology  are  quite  similar  for 
torpedoes  and  A  AMs.  Unfortunately,  the  estimates  of  the  parameters  of  the 
fighter  aircraft  model  are  too  “noisy”  to  be  much  use.  The  estimated  rate  of 
technology  growth  in  response  to  R&;D  expenditures  is  about  0.63,  which  implies 
that  if  R&D  grows  at  1  percent,  the  rate  of  technology  growth  will  be  about 
0.63  percent.  There  is  somewhat  greater  variability  among  the  estimates  of  the 
discount  rate  parameter  r,  but  neither  of  these  rates  (IT  or  23  percent)  seems 
intuitively  unbelievable. 

The  lesson  from  this  modeling  exercise  seems  to  be  that  it  is  easier  to  estimate 
the  parameters  of  a  model  for  more  generic  systems,  such  as  torpedoes,  than  for 
less  generic  systems,  such  as  fighter  airplanes.  If  a  modeling  approach  is  to  be 
used  for  aircraft  R&D.  it  probably  is  best  to  use  historical  data  for  a  similar 
kind  of  aircraft  to  estimate  a  model.  Nevertheless,  the  simple  control  model 
outlined  and  estimated  here  suggests  that  important  insights  may  be  gained 
from  a  modeling  approach. 


COMPARISON  OF  ALTERNATIVE 
APPROACHES 


It  appears  from  the  analysis  undertaken  in  this  paper  that  several  alterna¬ 
tive  approaches  may  be  acceptable  for  generating  projections  of  R&D  project 
cost /  time  profiles.  The  simplest  method,  averaging  historical  cost/time  profiles, 
works  best  for  systems  that  are  highly  generic,  such  as  torpedoes.  A  slightly 
more  sophisticated  approach  involves  constructing  a  cost/time  cumulative  dis¬ 
tribution  that  fits  the  observed  data  well  in  the  least-squares  regression  sense. 
However,  this  approach  also  works  best  for  systems  that  are  highly  generic.  The 
final  alternative  investigated  is  to  estimate  the  parameters  of  a  theoretical  cost 
minimization  model  using  historical  data,  and  use  these  parameter  estimates  to 
generate  cost  profiles  for  new  system  development.  This  third  alternative  is  not 
demonstrably  superior  to  the  other  two  in  terms  of  fit,  but  it  may  be  attrac¬ 
tive  when  projecting  cost/time  profiles  for  systems  for  which  there  is  no  close 
historical  analogue. 

As  additional  data  become  available,  it  will  be  interesting  to  estimate  struc¬ 
tural  models  and  compare  them  to  actual  experience.  If  it  turns  out  that  there 
are  stable  empirical  regularities  in  these  projects,  this  information  could  help 
rationalize  RizD  planning,  which,  after  all,  is  what  the  Quo  Vadis  effort  is  all 
about.  In  the  meantime,  the  project  staff  should  consider  some  kind  of  modeling 
approach  as  an  alternative  to  the  “first-round”  practice  of  using  the  most  recent 
(or  conceptually  most  similar)  system  development  project  as  a  model,  because 
using  any  single  project  as  a  model  for  all  subsequent  projects  of  a  similar  type 
is  not  an  efficient  use  of  historical  information. 
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